Papers with auto-regressive models
Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies have focused on auto-regressive models for generalization in named entity (NE) typing (NET) and recognition (NER) . however, little has been done in this direction for auto-Regressive LMs despite their popularity and potential to express a wide variety of NLP tasks in the same unified format. |
| Approach: | They propose to probe auto-regressive LMs for NET and NER generalization by resorting to meta-learning to assess the model's memorization of NEs. |
| Outcome: | The proposed model performs well on NET and NER generalization tasks, while relying more on NE than contextual cues in few-shot NER. |
Do dialogue representations align with perception? An empirical study (2023.eacl-main)
Copied to clipboard
| Challenge: | masked language models produce stronger correlations than auto-regressive models, but humans and models make different response selection mistakes. |
| Approach: | They propose to use spoken conversation as a model to measure human comprehension behaviour. |
| Outcome: | The proposed model outperforms the model which produces the strongest correlation with human responses. |
Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a critical task in information extraction that is not covered in recent benchmarks. |
| Approach: | They compare 13 auto-regressive models using prompting and 16 masked models using fine-tuning on 14 NER datasets covering English, French and Spanish. |
| Outcome: | The proposed models outperform auto-regressive models in English, French and Spanish on 14 NER datasets. |
Conditional [MASK] Discrete Diffusion Language Model (2025.emnlp-main)
Copied to clipboard
| Challenge: | Auto-regressive models excel in natural language processing but struggle to generate diverse text and lack controllability. |
| Approach: | They propose entropy-adaptive Gibbs sampling and entropic-based noise scheduling to counterbalance each model’s shortcomings. |
| Outcome: | The proposed framework outperforms baseline models and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation. |
An Iteratively Parallel Generation Method with the Pre-Filling Strategy for Document-level Event Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate event roles require a given generation order . parallel methods suffer from inadequate training and manifest zero accuracies on some event roles. |
| Approach: | They propose an iteratively parallel generation method with the Pre-Filling strategy to generate event roles in parallel to avoid order selection. |
| Outcome: | The proposed method outperforms other entity-enhanced models and achieves state-of-the-art performance on two public datasets. |
DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for speech generation are not efficient due to low information density of speech data. |
| Approach: | They propose a method to integrate discrete diffusion models into speech generation tasks . they propose to apply diffusion forward process while employing diffusion backward process . |
| Outcome: | The proposed model achieves comparable results to the auto-regressive baselines with significantly fewer decoding steps (50 steps). |
Let’s Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCR (2025.findings-acl)
Copied to clipboard
| Challenge: | Various fusion strategies have been explored for integration of large language models into multi-modal systems. |
| Approach: | They propose a framework for deep fusion decoding that integrates large language models into cross-modal text recognition systems. |
| Outcome: | The proposed framework surpasses cascaded methods in English and Mandarin, and significantly reduces WERs by 17.7%. |
Training compute-optimal transformer encoder models (2025.emnlp-main)
Copied to clipboard
| Challenge: | OptiBERT is a family of compute-optimal BERT-style models that matches or surpasses leading baselines while training with dramatically less FLOPS. |
| Approach: | They propose to train OptiBERT models with a Masked Language Model objective . they train a family of compute-optimal BERT-style models that matches or surpasses leading baselines . |
| Outcome: | The proposed model matches or surpasses leading baselines on GLUE and MTEB while training with dramatically less FLOPS. |